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ABSTRACT 

There  is  a  need  to  back  up  critical  timing  infrastructure  at 
the  national  level.  This  paper  describes  a  joint  project 
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employing  commercial  equipment  to  send  national  timing 
signals  through  a  telecommunication  network.  This 
experiment  connects  UTC(NIST)  in  Boulder,  Colorado 
with  UTC(USNO)  at  the  Alternate  Master  Clock  at 
Schriever  AFB  via  a  telecommunication  provider's  optical 
network  using  the  Precise  Time  Protocol  (PTP)  to 
compare  the  time  standards.  The  experiment  was  started 
in  April  2014  and  will  run  through  the  end  of  2014.  The 
paper  provides  insight  into  both  the  planning  and 
validation  of  the  transport  path  as  well  as  analysis  of  the 
experimental  data.  The  focus  here  is  using  a  US 
commercial  telecom  carrier  to  transfer  time  between  two 
national  real-time  standards  of  UTC.  While  many 
researchers  have  shown  that  fiber  can  transfer  time  and 
frequency  with  high  accuracy,  this  experiment  addresses 
the  practicality  of  using  the  US  telecom  infrastructure  for 
timing.  Our  results  thus  far  show  a  bias  of  about  40 
microseconds  between  the  two  one-way  directions  of  PTP 
signals,  with  the  best  method  having  a  variation  of  under 
about  50  nanoseconds  peak-to-peak.  Research  is 
continuing  to  determine  the  cause  of  the  bias. 

INTRODUCTION 

A  number  of  government  agencies  have  discussed  a  need 
to  back  up  critical  timing  infrastructure  at  the  national 
level  [1].  In  September  2011,  Century  link,  a  local 
Colorado  telecom  provider,  agreed  in  principle  to  a  two- 
year  experiment  linking  the  UTC  timescale  of  the 
National  Institute  of  Standards  and  Technology  (NIST)  in 
Boulder,  Colorado  and  the  US  Naval  Observatory 
(USNO)  Alternate  Master  Clock  (AMC)  at  Schriever 
AFB,  where  GPS  is  controlled.  The  US  Department  of 
Homeland  Security  (DHS)  issued  a  Request  for 
Information  (RFI),  Solicitation  Number:  RUKM2-A0009 
“Transferring  of  Time  via  Fiber  Network  Technologies,” 
in  December  2011,  requesting  information  on  how 
vendors  could  support  this  project  [2].  One  vendor, 
Symmetricom  at  the  time,  now  Microsemi,  gave  a 
detailed  plan.  A  three-way  Cooperative  Research  and 
Development  Agreement  (CRADA)  was  agreed  to  among 
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NIST,  Centurylink,  and  Symmetricom-Microsemi  and 
signed  in  January  2013,  to  last  until  December  31,  2014. 
We  are  currently  working  to  extend  this  past  December 
2014  to  December  2015.  The  goal  of  the  CRADA  was  to 
transfer  time  through  a  commercial  telecom  network  with 
an  accuracy  below  1  ps,  and  a  stability  below  100  ns. 

The  experiment  employs  the  Precision  Time  Protocol 
(PTP),  IEEE1588-2008  [3],  to  transfer  time  across  a 
public  telecom  network,  with  real-time  standards  of  UTC 
at  each  end:  UTC(NIST)  and  UTC(USNO).  This  has  not 
been  done  before,  to  the  best  knowledge  of  the  authors. 
Microsemi  is  providing  the  PTP  equipment  that  transmits 
and  receives  timing  signals  over  Gigabit  Ethernet  (GigE) 
[4]  on  optical  fibers.  The  fibers  run  from  the  two  national 
timing  labs  to  respective  Centurylink  offices,  where  the 
signals  are  multiplexed  into  their  network  on  a  specific 
optical  wavelength  that  is  not  shared  with  any  other 
customers.  The  experiment  has  used  two  different 
transport  methods.  First  has  been  to  transport  the  GigE  as 
a  SONET  [5]  payload  on  an  OC-192  [6]  system.  The 
second  has  been  to  use  the  Optical  Transport  Network 
(OTN)  [7]  system  to  transport  the  GigE  in  an  ODUO 
within  an  ODU2  transport. 

PTP  employs  two-way  time  transfer,  meaning  the  timing 
packets  are  sent  in  both  directions:  from  NIST  to  the 
AMC  and  from  the  AMC  to  NIST.  For  convenience  we 
refer  to  the  direction  from  NIST  to  the  AMC  as  forward, 
and  from  the  AMC  to  NIST  as  reverse. 

RESULTS 

First  we  discuss  the  PTP  over  SONET  results.  We  found 
an  asymmetry  of  40  ps  between  the  forward  and  reverse 
directions.  The  cause  is  currently  unknown.  In  addition, 
we  found  variations  in  the  one-way  delay  on  the  order  of 
300  ns.  These  were  approximately  deterministic  when 
nodes  were  timed  by  Cs  frequency  standards,  and  had 
more  random  wander  if  the  nodes  were  timed  by  GPS.  It 
may  be  that  the  variation  during  the  GPS  timing  has  a 
sinusoid  element.  These  results  are  illustrated  in  the 
following  plots.  Figure  1  shows  the  forward 
measurements  in  blue,  and  the  reverse  in  red.  There  is  a 
total  delay  of  about  2  ms  and  the  40  ps  asymmetry.  A  2 
ms  total  delay  at  the  speed  of  light  would  mean  a  distance 
of  600  km,  or  perhaps  400  km  in  fiber.  Given  that  the 
distance  between  the  two  in  a  straight  line  is  just  under 
200km,  it  becomes  clear  that  the  signals  must  be  buffered 
and  forwarded  by  equipment  in  the  path.  We  also  note 
that  variations  in  one  direction  are  somewhat  mirrored  in 
the  reverse  direction.  That  is,  a  slope  up  in  one  direction 
is  matched  by  a  slope  down  in  the  opposite  direction. 
However,  jumps  do  not  seem  to  be  matched. 

In  Figure  2  we  have  set  the  minimum  offset  of  each  plot 
to  0.0  from  both  paths  to  see  the  deviation  in  the 


measurements.  For  most  of  this  period  nodes  were  timed 
by  Cs.  clocks,  showing  a  slope  of  about  50  ns/d  with 
occasional  resets  of  about  300  ns.  A  period  in  the  middle 
is  marked  where  GPS  timing  was  used.  Here,  the  system 
accumulated  wander  with  no  clear  systematic  behavior. 
There  could  perhaps  be  a  sinusoid  effect. 

Following  these  results  we  switched  to  using  the  OTN  as 
the  transport.  There  were  two  reasons  for  doing  so.  First, 
we  wanted  to  begin  to  find  the  cause  of  the  40  ps 
asymmetry.  Changing  the  transport  was  accomplished 
simply  by  changing  the  card  that  encoded  the  GigE 
signals  into  and  out  of  the  Centurylink  network. 

Switching  to  OTN  would  allow  us  to  see  if  the  40  ps 
asymmetry  was  due  to  the  card  that  encoded  the  signal 
into  the  SONET  system.  Secondly,  we  wanted  to  see  if 
the  OTN  system  would  be  more  stable  than  SONET.  We 
show  plots  of  the  results  in  what  follows.  In  brief,  we 
found  that  the  OTN  data  were  much  more  stable,  but  that 
the  40  ps  asymmetry  remained.  Figure  3  and  Figure  4 
show  data  for  the  OTN  analogous  to  how  Figure  1  and 
Figure  2  show  data  for  the  SONET  system. 

In  Figure  3  we  see  with  OTN  a  similar  total  delay  and 
asymmetry  as  for  the  SONET  data,  but  even  here  we  can 
see  that  the  lines  appear  more  stable.  In  Figure  4,  we  set 
the  minimum  offset  of  each  plot  to  0  as  in  Figure  2,  and 
we  see  a  peak-to-peak  variation  of  50  ns  over  33  days. 
Part  of  this  is  an  apparent  trend  in  the  data.  In  the  short 
term,  the  stability  is  4  ns,  which  is  the  granularity  of  the 
PTP  measurement  system. 

If  we  subtract  the  forward  packets  from  the  reverse  and 
divide  by  two,  we  see  the  time  transfer  capability.  This 
combination  of  data  is  the  method  in  using  two-way  time 
transfer  data  for  cancelling  the  path  delay.  Figure  5 
shows  this  over  a  40  day  period.  We  see  a  peak-to-peak 
deviation  of  26  ns,  and  a  time  transfer  offset  of -19.1  ps. 
This  is  the  time -transfer  capability  of  this  system  if  used 
independent  of  any  other  time  transfer  system,  such  as 
GPS. 
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Symmetricom  TimeMonitor  Analyzer  (file=OC192_baseline-2014_04_16-1ppm-cumulalive.lwy) 

Phase  deviation  in  units  of  time;  Fs=15.74  mHz;  Fo=1 0.000000  MHz;  201 4/04/1 G  19:24:38 

Two-Way  Fwd/Rev  PDV  Phase;  Samples:  102492;  0C192  Baseline  Measurement;  MasterUUID:  00B0AEFFFE 


Figure  1 :  PTP  over  SONET  results  over  75  days,  showing  the  forward  delay  in  blue  and  the  reverse  in 
red.  The  total  delay  is  about  2  ms  with  about  a  40  jus  asymmetry. 


Symmetricom  TimeMonitor  Analyzer  (file=0C1 92_baseline-201 4_04_1 6-1  ppm-cumulative.twy) 

Phase  deviation  in  units  of  time;  Fs=15.74  mHz;  Fo=10.000000  MHz;  2014/04/16  19:24:38 
Two-Way  Fwd/Rev  PDV  Phase;  Samples:  102492;  OC192  Baseline  Measurement;  MasterUUID:  00B0AEFFFE 
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Figure  2:  Data  as  in  Figure  1  with  the  minimum  offset  of  each  plot  set  to  0.0  to  show  the  deviations. 

For  most  of  this  period  nodes  were  timed  by  Cs  clocks,  showing  a  slope  of  about  50  ns/d  with  occasional 
resets  of  about  300  ns.  A  period  in  the  middle  is  marked  where  GPS  timing  was  used.  Here,  the  system 
accumulated  wander  with  no  apparent  systematic  behavior. 
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Symmetncom  TimeMoniloi  Analyze!  (file=OTN_Baseline-2014_10_09  -20_31-1ppin_cuinulative.twy) 

Phase  deviation  in  units  of  time;  Fs=15.35  mHz;  Fo=10.000000  MHz;  2014/10/09  20:33:42 
Two-Way  Fwd/Rev  PDV  Phase;  Samples:  43922;  OTN  Baseline  Measurement;  MasteiUUID:  00BQAEFFFE02' 
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Figure  3:  PTP  over  OTN  results  over  33  days,  showing  the  forward  delay  in  blue  and  the  reverse  in  red. 
As  for  the  SONET  case,  the  total  delay  is  about  2  ms  with  about  a  40  jus  asymmetry. 


Symmetiicom  TimeMoniloi  Analyze!  (file*OTN_Baseline-2014_10_09--20_31Tppm_cumulative.tpk) 

Phase  deviation  in  units  of  time;  Fs=15.35  mHz;  Fo=10. 000000  MHz;  2014/10/09  20:33:42 

Two-Way  Fwd/Rev  PDV  Phase;  Samples:  43922;  OTN  Baseline  Measurement;  MasteiUUID:  00BQAEFFFE02' 


Figure  4:  PTP  over  OTN  data  with  the  minimum  offset  set  to  0.0  shows  a  peak-to-peak  variation  of  50 
ns  over  33  days.  Part  of  this  is  an  apparent  trend  in  the  data.  In  the  short  term,  the  stability  is  4  ns, 
which  is  the  granularity  of  the  PTP  measurement  system. 


217 


Symmetiicom  TimeMonitoi  Analyzei  (file«OTN_Baseline-2014_10_09-1ppm_cumulative.twy) 

Phase  deviation  in  units  of  time;  Fs=15.13  mHz;  Fo-10. 000000  MHz;  2014/10/09  20:33:42 
Two-Way  Normalized  Offsetlnv  Phase;  Samples:  52524;  Initial  phase  offset:  -19.0760  usee;  OTN  Baseline  M 
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Figure  5:  Time  transfer  capability  using  the  OTN.  The  value  of -19.1  ps  is  due  to  an  asymmetry  of  38.2 
ps.  The  peak-to-peak  deviation  is  26  ns,  with  the  short  term  no  more  than  4  ns. 


Symmetiicom  TimeMonitoi  Analyzer  (file=0C1 92_baseline-201 4_04_1 6-1  ppm-cumulative.twy) 

Phase  deviation  in  units  of  time:  Fs=16.35  mHz;  Fo=10.000000  MHz;  2014/04/16  19:24:38 

1  (blue):  Two-Way  Rev-Fwd  Phase;  Samples:  27909;  Start:  13267;  Stop:  41175;  Initial  phase  offset:  40.9360  usee;  0C192  Baseline  M 

2  (led):  Two-Way  Rev-Fwd  Phase;  Samples:  25835;  Stall:  218;  Initial  phase  offset:  40.5040  usee;  OTN  Baseline  Measuiement;  Maste 
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Figure  6:  The  asymmetry  from  the  reverse  minus  forward  directions  for  the  SONET-based  circuit  (blue) 
and  the  OTN-based  circuit  (red)  plotted  on  the  same  graph.  The  two  systems  have  a  very  similar 
average  asymmetry. 
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DIAGNOSTICS 

We  decided  to  pursue  the  cause  of  the  40  ps  asymmetry 
by  breaking  the  circuit  into  sections.  The  path  from 
NIST,  Boulder  to  the  AMC  at  Schriever  AFB  was  chosen 
to  have  three  segments,  by  breaking  it  in  a  Denver  office 
and  in  a  Colorado  Springs  office.  The  plan  is  to  do  PTP 
time  transfer  from  both  NIST  and  the  USNO  AMC  to 
each  of  these  offices.  This  will  require  the  use  of 
additional  equipment,  as  we  will  need  PTP  masters  in 
each  of  these  central  offices  as  well  as  a  UTC  reference. 
For  the  UTC  reference  we  will  use  UTC  from  GPS. 
Comparing  each  realization  should  allow  an  uncertainty 
in  the  references  of  no  more  than  a  few  10’s  of  ns,  i.e. 
comparing  UTC(NIST),  UTC(USNO)  at  the  AMC,  and 
UTC(USNO)  as  transmitted  by  GPS.  This  should 
certainly  allow  us  to  find  which  segments  contribute  to  40 
ps  offset.  We  are  currently  waiting  for  the  installation  of 
the  needed  equipment. 

In  the  meanwhile,  we  performed  a  number  of  loopback 
tests  from  NIST  to  these  locations.  Note  that  the 
loopback  was  actually  a  loop-back  of  the  two  directions 
individually,  i.e.  the  forward  and  reverse  directions  each 
went  from  one  port  of  the  NIST  PTP  device  out  and  back 
to  another  port  of  the  same  device.  This  method  was 
unable  to  detect  any  one-way  asymmetry,  as  it  would 
cancel  in  the  loop  back.  What  we  were  able  to  measure 
was  an  asymmetry  in  the  initial  hardware  that  converts  the 
GigE  to  an  ODUO  and  vice  versa.  The  manufacturer  was 
able  to  confirm  that  these  devices  have  a  random 
asymmetry  of  up  to  3  ps  that  cannot  be  controlled.  Since, 
in  the  circuit  between  NIST  and  the  USNO  AMC,  there  is 
one  of  these  devices  serving  each  end,  this  could  account 
for  up  to  6  ps,  but  not  40  ps.  When  the  loop-back  circuit 
that  goes  through  only  one  conversion  device  is  brought 
up,  measured,  then  released  and  re-created  and  measured 
again,  we  do  indeed  see  variations  of  no  more  than  3  ps. 
This  explains  why  the  total  asymmetry  in  Figure  5  is 
approximately  38  ps,  while  in  Figure  1  and  Figure  3  it  is 
about  40  ps. 

The  next  step  for  us,  as  of  late  November  2014,  remains 
to  measure  the  asymmetry  in  one-way  delays  in  segments 
of  the  circuit.  As  much  as  possible,  we  hope  to  determine 
the  source  or  sources  of  this  asymmetry.  Of  particular 
interest  is  the  fact  that  the  asymmetry  is  very  close 
between  both  the  SONET -based  circuit  and  the  OTN- 
based  circuit.  Figure  6  illustrates  this,  showing  the 
reverse  minus  forward  one-way  delays  (i.e.  these  are  not 
divided  by  two,  as  is  done  for  the  time  transfer 
illustration).  Although  the  OTN  circuit  (red  curve)  is 
much  more  stable  than  the  SONET  one  (blue  curve),  the 
mean  asymmetry  for  the  two  methods  seems  very  close. 
Recall  that  there  would  be  a  variation  in  the  asymmetry 
up  to  6  ps,  i.e.  up  to  3  ps  at  each  end,  due  to  having  taken 


the  circuit  down  from  the  SONET  system  and  then 
brought  it  back  up  for  the  OTN. 

CONCLUSIONS 

While  we  have  not  found  a  time  -transfer  accuracy  below 
1  ps,  with  the  OTN  system  the  stability  is  well  below  100 
ns.  If  one  imagines  a  partial  backup  to  GPS  timing, 
where  GPS  can  be  used  to  calibrate  the  asymmetry,  and 
where  PTP  is  available  for  when  GPS  is  unavailable,  then 
it  appears  that  this  OTN  system  would  support  better  than 
100  ns  time  transfer.  If  for  any  reason  the  circuit  is  lost 
and  re-created,  one  would  need  GPS  or  some  alternative 
time  reference  to  calibrate  the  new  asymmetry. 

We  look  forward  to  further  study  of  this  circuit  to  learn 
more  about  the  source  of  the  approximately  40  ps 
asymmetry.  The  long-term  plan  is  to  extend  this  research 
to  other  circuits,  perhaps  spanning  the  continental  US,  and 
perhaps  establishing  standards  supporting  such  time- 
transfer  systems. 

DISCLAIMERS 

This  is  a  contribution  of  the  U.S.  government,  hence  is 
not  subject  to  copyright. 

Any  mention  of  company  or  equipment  names  is  for 
information  only  and  not  meant  is  an  endorsement. 
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